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A simulation study was designed to assess the 
severity of regression effects when a set cf selection scores is also 
used as^retest scores as this pertains to BMC Model A of the 
EleBent»y and Secondary Education Act Title I evaluation and 
reportilq syste*. Data sets were created with variou* characteristics 
(varying data reliability and extreaeness of sukgrcups) that are 
relevant to the regression phencaenon. These data sets were analyzed 
to obtaiii indices of the aiount of regression which Bight occur under 
various conditions. An adjustment lethcd was presented which 
predicted pretest scores on the basis cf selection scores and the 
correlation between selection and pretest, the correlatict 
coefficient being replaceable by such Indices of reliability as the 
internal consistency coefficient. Results suggest that when data 
reliability is high (.9« or above) the iipact of refression effects 
appears saall; but when reliability is low the effects of regression 
a^e hard to predict, when selection test scores are of high 
reliability, such regressed scores aay be used as a pretest aeasure 
without severely biasing the evaluation results. Results can be used 
to formulate a rule^of-t huab for adjusting selection scores when they 
are to be used as pretest scores also. (Author/CF) 
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CAN GELiJcTION TESTS BE USED AS PRETEST^S? 

* Kim Onn Yap ^ ' • 

' i ■ 

; Northwest Regional Eduicational Laboratory 

INTRODUCTION / 

MocTel A bf the proposed ESEA Title I evaluation and reporting system (Tallmadge & 

/ ■ ■■ ■' V 

Wood, ll<^6; Tallmadge, 1976) specifies that independe^il? measures shoiildJt)e used foi\ 
selection and pretest to eliminate regression effects due to measurement ei4or. These 
effects are expected to 'occur when extreme group5:.(e. g^, the bottom 20 percent of the 
school population) are selectdfc to participate in Title I programs. The size of regression 
is a function of the correlation between the^^wo measures Jnvolve<J, i. e. , the correlation 
between selection and pretest. A second paramftter which affects the size of regression 
is the difference b^t^l^en the subgroup means. That is, the greater tb^^^meanjdifference 
the more regression ^(Campbell and Erlehacher, 19T0L. 0,* 
The observed sfcone is an unbiased estimate! of the^-trtie score only for unaelect^d 

■ \ ' ■ ~T~ 

scores, that is, for total sets of scores th^have been allowed to fall freely. \\^>ew:B 
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scores have been selected because- of theil^bserved values, they become biased estimates 
of true scores in the direction indicated by regression toward the population mean. 

ll^e specifically, the regression phenomenon stems from the behavior of J 
measurement error. Positive measurement error contributes predominantly to above- 
average test scores; negaif^ve measurement error contributes predominantly to below- 
average test scores. A higher scorj|-4^ more likely to contain a positive error than 
a lower score, and vice ver^^t. While error can be expected to be randomly distribi^d 
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across scores for the total group, its means are necessarily not equal to zero for low and 
high scoring subgroups. 

The regression effect is removed when the low-ficoring subgroup is retested, error 
being again randomly distributed across the group members. Negative error and 
positive error cancel each other, yielding an expected mean error of zero. This 
results in retest scores that a^e higher than the first set of scores which contained a 
preponderance of negative error. The higher retest scores are^clQser to the mean of 
the total group and also closer to the true scores. 

The regression effect poses a problem for Model A in that the difference between 
the retest scores and thr first set of test scores could mistakenly be interpreted as 
an achievement gain — when no real difference in achievement exists. Tt follows that if 
the first set of test scores is used as a pretest measure and the second a posttest measure, 
we will obtain a biased estimate of achievement gain. It further follows that if the first 
set of scores is used for selection purposes, it will be necessary to retest the 

s 



selected subgroups to get an unbiased pretest measure. A third set of test scores wil 



need to be collected to provide an unbiased posttest measure. 

THE PROBLEM AND PURPOSE OF THE STUDY , 
^n Title I schools, low-scoring studentS^are typically selected for participation in 
Title I projects. This is an appropriate procedure for identifying low-achieving students — 
although some students not selected may have lower true achievement levels tlfen some 
who are selected (Tallmadge, 1976). The observed scores of the selected st;)idents will, 
however, con^i^ a preponderance of negative measurement error. If used as pretest 
scores^ these obs^jrved scores will provide b biased estimate of the students^ initial 



achievement level ajid, thei^efore, represent a biased basis fo^j estimating achievement gain 
at posttest. The solution, as indicated earlier, is to obtain a second set of test scores to 
serve as the pretest measure. 2 >l ^ ^ , ; 



Unfortunatelifc for a number of reasonsi this solution is not always available to 



I 

Title I project adtnMstrators. Such factors as school scl>edule, b&dget and time 

constraints tnot to mention the burden of overtesting students) often' make It impractical 

# ■ ■" 

or impossible to obtain a second set of test scores as the pretest measure. In other 
words, the same set of test scores will necessarily have to be used as the selection 

J 

criterion as well as the pretest measure. 

This points out a need to go beyond explaining that regression bias exists In 
those situations. We need to estimate the amount or severity of the bias when the 
selection test is also used as the pretest. In addition, there Is a need to assess the 
performance of any statistical procedures that may exist for reducing or eliminating 

the bias. ^ 

- 

The purpose of this study was therefore twofold. First, th^^lnvestigatlon was 
au attempt tu estimate the severity of regression bias when a seleptlon measure is also 

K i 

used as the pretest measure. Secondly, the study was aimed at^^s^sessing the 
performance of existing procedures for reducing or eliminating the bias. The study 
essentially consisted of a simulation of data sets resembling those typically encountered 
In Title I evaluation and an analysis of the simulated data. It was hoped that s^nrie 
practical guidelines could be deVe loped which would suggest conditions under which 
a selection measure may also be used as th^^retest measure. \ 

the follqwing formulae which defined 

selection and pretest scores: 



THE SIMULATION 
Data sets were generated by usiif^ 



Xj^lj Tlj + Eij, and 
Xgijf Tlj +^E»ii, 



I 

where Xj^lj Is the selectl^ji score of person j In group I, Xgij Is the jjretest score of 
person j in group i, Tij is the true score of person j in group'i, and Eij and E'ij are' | 
error scorfes at selection and pretest times, respectively, of person j tn\roup i, 

(i = 1, , 100; j= 1, , 100). 

A computer program was developed to generate values for the simulated 
selection and pretest scores. These values were derived from random numbers 
provided by Subroutine GAUSS (IBM, 1968), For example, a pair of selection and 
pretest scores may be obtained«.s follows: 

Tij=.9(Ni), ' 

Eij = .1 (Ng), and 

E'ij^.MNg), / 

where the N's are two-digit random numbers provided by GAUSS. 

Subroutine GAUSS generates normally distributed random numbers with a given 
mean and standard deviation which were chosen to^l^^SO and 21. 06, respectively, for 

- ' - \ 

this study. They are also the mean and standard deviation of th^ normal curve 
equivalen!^(NCE) scores. 

\ 

It should be obvious that by <rarying the multiplii^rs of tli6 random numbers <^ne 
couTd manipulate the reliability of the data. In the above exajnple we have'se^ the 
true score (Tij) and error score (Eij) to be nine-tenths and one-tentii, respectively, 
of the observed selection score (X^ij).\^The same is true of the observed pretest score.^ 
other words, the ratio of the true score to the error score in terms of size is 
9:1 for this pair of selection and pretest scores. ;li|we-tiad set Tij = .8 (N^), Eij = 

" ^^^^ 

. 2 (N ) and E' ij .2 (N3) we would have had larger ^rrcjra and le^ss reliability for 

^ '\ ■ 

Xjij and X^ij . The size of the true score relative tb the error score was described as 

1 

the error ratio in this study. f% ' ^ 



Four error ratios were used In generating the data sets: 9:1, 8:2, 7:3 and 6:4, 
This means that data of varying levels of reliability were gjK^ulatecl. More specifically, 
100 data sets (each consisting of 100 simulated cases) were created for each of the 
four error ratios. The entire simulatiou thus include 400 data sets with a total of 
40,000 hypothetical cases. Th^ sample means of^hese data seii^^ range from 49,86 to 

^- . i' ■ . . 

50.51, closely approximating thd^p^pulation mean'of 50.\ The standard deviations, 
however, tend to be generally smaller than the population standard deviation of 21.06, 
ranging from 15. 26 to 19. 03. In all ^^es, selection and related pretest 



scores 



are shown to llave approximately the same-meail^nd standard deviation^ Sample means 
and ^standard deviations of the data s^ts ^r^*1'eporte<J in\ Table 1. 



- . TABLE r^^^^^^ 

Ov^rall^^eans and Standard Deviations of the Simulai^Data Sets* 



Error Ratio 


Mean 


Standard Deviation 




Selection 


Pretest 


Selection 


Pretest 


9 = 1 \ 


50.51 


5j3.49 


19.03 


19.02 


8:2 


50.14 


50.10 


17.41 


17.38 


7:3 


» 49.86 


49.86 


15.92 


16.05 


6:4 


49.96 . 


50.04 


15.26 


15.27 



♦These fneans and standard i^eviations were each based on 100 data sets each consisting 
of 100 simulated cases. / 



A word should be said about th'fe relationship between wifat was described as the 
error ratio and reliability coef f lent as the' term is used in the measurement literature. 
As defined by Gulliksen (1950), the reliability' coefficient is the ratio of tru^ariance 
to observed variance. Since multiplying an element by a constant will multiply the 
variance by the square of the constant, the ratio of true variance to error variance 
will be greater than the error ratio. Thus, error ratios should be considered as\ 
lower bound reliability coefficients. That is to say, an error ratio pf 9:1, for example, 
represents a reliability coefficient which is in fact higher than . 90. 

As indicated earlier, the amount of regression is a direct function of the 
correlation coefficient between selection and pretest. In standard score from (i.e. , as 
Aivintions from the means divided by their standard deviations) on^ can predict a 
pretest score on the basis of the selection score and the correlation coefficient 
between s^election and pretest as follows: 

X2 = rxj^, where 
XI is a selection score, ^2 indicates a predicted (or adjusted) pretest score, 
and r is the correlation coefficient between selection and pretest for the unselected group. 
Making certain assumptions about the means and variances of the selection and pretest 
scores (i.e. , both have the same mean and variance for the unselected group) it can 
be shown, in raw score form, that 

A. — — 

X2 ^ Xj + r(X^ - Xj), where ^ 

_ A 
Xj^ is a selection raw score, X j is the selection mean score, X2 is a predicted -x 

pretest score, and r is the correlation coefficient bet^veen selection and pretest 
scores for the unselected gr'g^tp. 
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The present study used this adjustment method to predict pretest scores and' ' > 
aJlessed its performance with data of various reliability levels. 

ANALYSIS AND RESULTS 

Two types of analyses were performed on the data. First, mean differences * 

between selection and pretest scores were computed for each of the data sets. These 

mean differences were then summed and averaged for each of the data types (i.e., data 

sets of different levels of reliability) as an overall index of the regression effect. 

Standard deviations were computed to indicate the distributions of these mean 

differences. Secondly, mean differences betiveen pretest scores and predicted pretest 

* 

scores were computed for each of the data sets. Again, these mean differences were 
summed and averaged for each of the data types as an overall index of the adequacy of 
the adjustment method. Standard deviations were computed to Indicate the distributions 
of the mean differences. 

The analyses were performed for the total sample and three subsamples 
representing extreme subgroups. These subsamples consisted of the bottom 30, 
20 and lO'percent of the c^ses in the total sample. Results of the analyses are presented in 
Tables 2 and 3. 

/ 
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TABLE 2 

Mean Differences Between Selection and Pretest* 



Sample 


Error Ratio** 


Mean 
Difference 


Standard Deviation 
of Difference 


Total ^ 


9:1 

(r = .99) 


.02 


.27 


8:2 

(r = .94) 


.04 


.57 


7:3 

(r = .84) 


-.00 


.96 


6:4 

(r - . 70) 


-.07 


1.30 


Bottom 30% 


9:1 


-. 24 


.58 


8:2 


-1.23 


1. 07 


7:3 


-2.66 


1. 64 


6:4 


-5.51 


2.21 


Bottom 20% 


9:1 


-.32 


.70 


8:2 


-1.53 


1.47 


7:3 


-3.25 


1.91 


6:4 


-6.37 


2.61 


Bottom 10% 


9:1 


-.38 


.97 


8:2 


-1.72 


2. 04 


7:3 


-4.35 


2.68 


6:4 


. -8.06 


3.27 



♦f^ach mean difference was based on 100 data sets. 

♦♦Error ratios should be considerec\ as lower bound reliability. In other words, an 
error ratio of 9:1 represents a reliability coefficient higher than .9(^r The average 
correlation coefficients between selection and pretest for the total samj^les are in 
parentheses. 

r • 



10 

8 , 



TABLE 3 

Mean Difforerices Between Pvmest and Predicted Pretest* 



Sample 


Error Ratio** 


Mean ■ 
Difference 


Standard Deviation 
of Difference 


Total 


9:1 

(r = .99) 


-.02 


.27 


8:2 

(r - .94) 


-. 04 


.57 


7:3 

(r = .84) 


. 00 

— ^ ; . 


.96 


6:4 

(r = .70) 


.07 


1.30 


Bottom 30% 


9:1 


-.03 


.57 


8:2 


. 05 


1. 03 


7:3 


-.20 


±1-5^ 


6:4 


. 16 


2. 03 


Bottom 20% 


9:1 


-.0(T 


.69 


8:2 


. 12 


1.43 


7:3 ^ 


-. 20 


»1 75 


6:4 


-.10 


2.30 


Bottom 10% 


9:1 


-.02 


.95 


8:2 


-. 04 


2.00 


7:3 


.06 


2.44 


6:4 


-.02 


2.97 



♦Each mean difference was based on 100 data sets. 

♦♦Error ratios should be considered as lower bound reliability. In other words, an error 
ratio of 9:1 represents a reliability coefficient higher than .90. The average correlation 
coefficients between selection and pretest for the total samples aife in parentheses. 
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The same analyses were re^»ated on the absolute mean differences for each of 

the data se^s. These analyses were performed to assess the magnitude of mean 

k 

differences, regardless of the direction (positive or negative) of the differences, between 
seh^ction and pretest scores and between-pretest and predicted pretest scores. As would 
be expcM'ted, random fluctuations yielded a numt)er of instances in which the selection 
mecm scy?re was actually higher than the pretest mean score. For the same reason, 
the pr(ulict(-d pret(»st mean score was higher than the pretest mean score in some cases 
and lower in others. Hesults of the analyses on absolute mean differences were 
summarized in l'ai)les 4 and T). 
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TABLE 4 

Mean DLftertjnces Between Selection and Pretest Based on Absolute Values* 



Sample 


Error Ratio** 


Mean 
Difference 


Standard DeviatLon 
of Difference 


Total 


9:1 

(r-.99) 


.22 


. 16 


8:2 

(r = .94) 


.47 


. 32 


7:3 

(r - .84) 


. 75 


. 60 


6:4 

(r . 70) 


1. 07 


. 74 


Bottom 30% 


9:1 


.51 


.36 


3/2 


1.35 


.92 


7:3 


2.74 A 


1.51 


6:4 


5.54 


2. 12 


Bottom 20% 


9:1 


. 62 


.46 


8:2 


1.73 


1.23 


7:3 


3.27 


1.87 


6:4 


6.38 


2. 58 


Bottom 10% 


9:1 


.85 


. 59 


8:2 


2.20 


1.50 


7:3 


4.46 


2.50 


6:4 


8.06 


3.27 



*Each mean difference was based on 100 data sets. 

**Error ratios should be considered as lower bound reliability. In other words, an 
error ratio of 9:1 represents a reliability coefficient higher than .90. The average 
correlation coefficients between selection and pretest for the total samples are in 
parentheses. 
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TABLE 5 

Mean Differenct s Between Pretest and Predicted Pretest Based on Absolute Values* 



Sample 


Error Ratio** 

^ ( 


I^ean 
Difference 


Standard Deviation 
of Difference 


Total 


9:1 
. (r - .99) 


. 22 


. 16 


(r = .94) 


.47 

■ ■ 


. 32 


i :o 

(r = .84) 


.75 


. 60 


6:4 

(r - . 70) 


1. 07 


.74 


Bottom 30% 


9:1 


.45 


.35 


8:2 


4^ .83 


. 61 


7:3 


1.21 


.92 


. 6:4 


1. 61 


1.24 


Bottom 20% 


9:1 


.54 


.42 


8:2 


1. 15 


.84 


7:3 


1.42 


1. 02 


6:4 


1. 78 


1.44 


Bottom 10%, 


9:1 


.78 


. 55 


8:2 


1.56 


1.24 


7:3 


1.95 


1.47 


6:4 


2. 36 


1. 78 



*Each mean difference was based on 100 data sets. 

♦♦Error ratios should be considered as lower bound reliability. In other words, an error 
ratio of 9:1 represents a reliability coefficient higher than .90. The average correlation 
coefficients between selection and pretest for the total samples are in parerttheses. 




In looking at the results summarized in Tables 2 to 5, it is elear that, overall, 
the regression bias does exist. It is also elear that the size of sueh bias is a direet 
funetion of data reliability and the " tijxtremeness " of the subgroup. Thus, relatively 
small regression effeets were found with respeet to data sets of high levels of 
relTability (e.g. , data sets with error ratios of 9:1 and 8:2). As shown^^^able 2, 
the mean differenees range from . 04 to -1.72 for the total sample and^ all subsamples. (The mean 
differences reported for the total sample may best be regarded as a result of normal 
random fluctuations rather than regression.) As the level of reliability drops (e.g. , in 
data sets with error ratios of 7:3 and 6:4) the impact of regression increases, with 
the mean differenees between selection and prett^st ranging from -2. 66 to -8. 06 for 
the subsamples. 

It is interesting to note that as the error ratio drops from one level to the next, 
the regression effeets tend to double in size. \Take the bottom 30 percent sutgroup for 
example. WTien the error ratio drops from 8:2 to 7:3, the mean difference increases 
in size from -1.23 to -2.66. Similarly, when the error ratio declines from 7:3 to 6:4, 
the mean difference increases from -2.66 to -^6.^3.. This pattern of results is observed 
in all the subsamples. 

The standard deviations for the mean differences appear to be relatively large, indicating 
great dispersion in the distributions. Confidence intervals for the mean differences are 

thus fairly wide vVhich suggests caution in using any guidelines which might be formulated 

\ 

on the basis of these results. 

As described earlier, predicted pretest scores were obtained on the l>asis of 
selection scores and the correlation coefficient between selection and pretest for the 
unselected group (i.e. , the total sample). Results In Table 3 suggest that this procedure 



O 13 I fr 

ERIC 



works remarkably well w^h all the data sets. The mean difference? between predicted 
and actual pretest scopes range from -.20 to . 16 for all the subsamples. The standard 
deviations, however, appear relatively large, suggesting sub^^?tntial random 
fluctuations in the mean ^merences. 

It should be noted that the correlation coefficient between selection and pretest is, in 
fact, a measure of reliability of the selection and pretest scores. It is equivalent to a 
test-retest reliability since the time interval between selection and pretest is assumed > 
to be short. Thus, any appropriate reliability coefficient (e.g. , internal consistency 
coefficient) could substitute for the correlation coefficient in the adjustment formula. 
This substitution makes the use of ttaj^djustment procedure possible in cases where 
selection seoroe are used as the pretest measure and a separate pretest is not given. 

Results of the analyses on absolute values (as displayed in Tables 4 and 5) provide 
some further insights. First, the similarity between the two sets of results in Tables 
2 and 4 suggests that while the pretest mean score is not always higher than the selection 
mean score, exceptions are few. Furthermore, when the selection mean score does 
turn out to be higher than the pretest mean score, the mean difference tends to be 
relatively small. (This is confirmed by a perusal of the raw data.) Secondly, the 
dissimilarity between the two sets of results in Tables 3 and 5 IndicateS that while the 
adjustment procedure can, overall^ be expected to work well with most data types, the 
accuracy of prediction is less than what the mean differences displayed in Table 3 would 
suggest. Furthermore, the accuracy of prediction appears to be a direct function of data 
reliability~-as one would expect. Judging from the results in Table 5, it would appear 
that with data of relatively high reliability (e.g., data with error ratios of 9:1 and 8:2), the 
adjustment procedure could/b^dqpended vpon to yield satisfactorily accurate predicted 

pretest mean scores, vylth absolute mean differences ranging from .45 to 1.56 for all the 
subsamples. 

o 16 
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SUMMARY AND CONCLUSIONS ^ 

We have attempted In this investigation to assess the severity of regression effects when 
a srt of selection scores is also used as pretest scores as this pertains to Model A of 
the Title I evaluation and reportLng system. We have created data sets with various 
characteristics (i.e., data reliability imd extremeness of subgroups) that are relevant to the 
regr(»ssion phenomenon. These data sets were analyzed to obtain indices of the amount Of 
regression which might occur under various conditions. An adjustment method was 
presented which predicted pretest scores on the basis of selection scores and the 
correlation coefficient botween selection and pretest, the correlation coefficient being replace- 
able by such indices of reliability as the internal consistency coefficient. 

Before discussing the implicatiorife of the results, it is necessary to point out 
some important limitations of the study. First, we have considered in this study only 
regression bias attributable to measurement error as it affects extreme subgroups. 
Other regression artifacts, such as those attributable to non-equivalency of control 
groups or differential growth rates for treatment and control groups (see, for example, 
Thorndike, 1942; Campbell & Erlebacher, 1970; Kenny, 1975; Campbell Si Boruch, 1975; 
Bryk & Weisberg, 1977) were not examined. Consequently, while we would recommend 
the use of the adjustment method when its use is appropriate, it is obvious that such 
adjustment does not remove the other forms of regression bias which may occur with the 
use of Model A. 

Secondly, we have studied only the impact of regression bias on the extreme 
low-scoring subgroups (which typically get selected to participate In Title I programs). 
Although one could quite confidently hypothesize that the impact of regression effects 
on the extreme high-scoring subgroi4)s would, in terms of size, be similar to that 




^' reported for the low-scoring subgroups, empirical testing of^that hypothesis goes beyond 

the 39op^ of the present study. 

^.^^^ Overall, the results of this simulative study suggest that when data reliability 
is high (e.g. , when reliability coefficients are .94 or above) the ii\ipact of regression 
effects apifcars small and perhaps negligible^in most cases. The use of the adjustment, 
procedure further reduces the size of the regression bias. This implies that when selection 
scores are obtained with an instrument of high reliability, such Scores— with or even without 



♦ adjustmt'nt--could be used as a pretest measure without severely biasing the evaluation^ 
results. This finding appears to ho^ regardless of how extreme the subgroup happens to be. 

When data reliability is rel^iiveijr low (e.g., when reliability cpefficients fall 
between .84 and .70) the use of tlie adjustment procedure is likely to reduce the amouut 
of bias to such a degree that the selection scores could still be used as a pretest measure 
without severely distorting the evaluation results. This i!s especially true of less jextreme 
subgroups (e.g. , subgroups formed by the bottom 20 or 30 percent of the group raembars) 
where the mean differences between pretest and predicted pretest are, in most capes, 
negligible. For subgroups formed by the bottom ten percent of the group members, the 
meim differences could, however, be rather substantial. 

The findings of the study offer another option (besides using Ihteadjustment method) 
to users of Model A. The results summarized in Tables 1-5 could refused to formulate 
a rule-of-thumb for adjusting selection scores when they are al^ to be used as pretest 
scores. More specifically, the mean differences reported^or the various data types and 
extreme subgroups could be used as approximate indices of T^^^ which had actually occurred 
in real data sets resembling the simulated data. These indices could then be used to adjust 
the means of selection scores accordingly. One could, for exampl^, add the appropriate 
mean differences to selection mean scores and use the adjusted sr^USctfon means as pretest 



( 
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means. Altet^tively, one could subtract the at>prof)riate mean differences from 
mean gaiiY scores when such gains have been computed with unadjusted selection 
scores serving as pretest seorey. We would hasten to add, however, that the suggested 
rule-of-thumb should be used only as a last resort iii/asmuch as it yields only a crude and 
indirect estimate of the regression bias which might 4aave occurred in the real data. 

It should also be cautioned that data sets uled in the present study have been created 
with normal distributions of selection and pretest scores. To the extqnt that a set of | 
real data consists af*€election or protest scores that lack normality in distribution (such 
as when group meni^ei-s are rank ordered and those with lower ranl^s are selected to form the 
treutfnmit group) the size of the regression bias may differ from those 

reported in this paper. Based on data derived from unj^ri^ly distributed random members, 
preliminary evidence suggests that the regression bias in those cases'^. e., when score 
distributions lack normality) will tend to be more substantial. * 
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